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DESCRIPTION 

IMAG E MATCHING SYSTEM USING THREE-DIMENSIONAL OBJECT 
MODEL, IMAGE MATCHING METHOD. AND IMAGE MATCHING 
PROGRAM 

< 



5 TECHNICAL FIELD 

The present invention relates to an image matching system using a 
three-dimensional object model, an image matching method, and an image 
matching program. In particular, the present invention relates to an image 
matching system capable of retrieving a reference image stored in a 

1 0 database (DB) on the basis of an input image of an object (human face) 

picked up under a different pose and illumination condition, and its image 
matching method and image matching program. 

BACKGROUND ART 
An example of a conventional image matching system is described in 

1 5 Tsutada etc., "Dictionary configuration method for discriminating person 
independently of face direction," The Transactions of the Institute of 
Electronics, Information and Communication Engineers, D-II, Vol. J78-D-II, 
No. 11 (1995), pp. 1639-1649 (hereafter referred to as first conventional 
technique). As shown in FIG. 26, an image matching system according to 

20 the first image matching system includes an image input section 10, an 
image matching section 40, a result display section 80, a reference image 
storage section 70, and a reference image registration section 75. 

The conventional image matching system having such a 
configuration operates as hereafter described. 

25 Reference images of various objects (such as reference face images of 

persons) picked up are previously stored in the reference image storage 
section 70 by the reference image registration section 75. However, 
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reference images greatly change depending on the condition at the time of 
imaging (conditions such as the pose and illumination). With respect to one 
object, therefore, a plurality of (a large number of) images imaged under 
various conditions are previously stored. 

5 The image input section 10 is implemented by using, for example, a 

camera. The imaged input image is stored in a memory (not illustrated). The 
image matching sectibn 40 compares the input image obtained from the 
image input section 10 with each of reference images obtained from the 
reference image storage section 70, calculates similarities (or distance 

10 values) of respective features, and selects a reference image having the 
greatest similarity (or the shortest distance) for each object. Each image is 
represented by gray level features. In the calculation of similarity between 
features and the calculation of the distance value, for example, the 
normalized correlation and Euclidean distance are used. The result display 

1 5 section 80 displays a reference image of an object having the greatest 

similarity selected from among the reference images, as a matching result 
(or displays candidate reference images in the descending order of the 
similarity). 

Another example of a conventional image matching system is 
20 described in Japanese Patent Application Laid-Open No. 2000*322577 

(hereafter referred to as second conventional technique). As shown in FIG. 
28, the conventional image matching system includes an image input 
section 10, an image conversion section 35, a partial image matching section 
45, a result display section 80, a reference image registration section 75, a 
25 representative three-dimensional object model storage section 20, and a 
three-dimensional object model registration section 25. 
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The conventional image matching system having such a 
configuration operates as hereafter described. 

One or more representative three-dimensional object models 
obtained from the three-dimensional object model registration section 25 are 
5 previously stored in the representative three-dimensional object model 
storage section 20. As regards a partial region common to the input image 
obtained from the image input section 10 and each of the reference image 
obtained from the reference image storage section 70, the image conversion 
section 35 converts the input image and/or the reference image so as to 
1 0 make the input condition (such as the pose condition) the same by using a 
three-dimensional object model obtained from the representative three- 
dimensional object model storage section 20, and thereby generates partial 
images. 

For example, as shown in FIG. 29, the partial region is a feature 
1 5 portion such as an eye, a nose or a mouth. By previously specifying a feature 
point with respect to each of the images and the three-dimensional object 
models, correspondence can be taken. The partial image matching section 45 
compares the converted input image obtained from the image conversion 
section 35 with a partial image of each of the reference images, calculates 
20 respective average similarities, and selects a reference image having the 
greatest similarity for each object. The result display section 80 displays an 
object having the greatest similarity among the reference images, as a 
matching result. 

DISCLOSURE OF INVENTION 
25 The above-described conventional techniques have various problems 

described hereafter. 
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First, the above-described first and second conventional techniques 
have a problem that a large number of reference images of an object to be 
registered, picked up under various conditions become necessary. 

The reason is as follows : the input image is directly compared with a 
5 reference image; if the input image pickup condition is not restricted, 
therefore, it is necessary to previously prepare reference images that are 
close to the input image in pickup condition in order to cope with a large 
number of pose and illumination conditions. As a matter of fact, however, 
there are infinite possibilities in the pose and illumination conditions, and it 
10 is practically impossible to previously prepare a large number of images 
associated with various conditions. 

Secondly, in the second conventional technique, the input image or 
the reference image is converted so as to square them with each other in 
pose, and comparison is conducted. If the number of the reference images of 
1 5 a three-dimensional object model is not sufficient or the pose is largely 

different, therefore, distortion caused by the conversion becomes large and 
matching cannot be conducted correctly, resulting in a problem. There is 
also a problem that it is very difficult to square the illumination conditions 
with each other by conducting conversion and a common region certainly 
20 needs to be present because images are compared in the common region. 

Thirdly, the conventional techniques have a problem that it takes a 
considerably long time to conduct matching. 

The reason is as follows" in the conventional techniques, the input 
image is compared with a plurality of reference images of respective objects; 
25 if the number of objects is M and the number of reference images of each 
object is L, therefore, image comparison must be conducted at least LD M 
times. 
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An object of the present invention is to provide an image matching 
system, an image matching method, and an image matching program that 
make it possible to retrieve a reference image registered in a database on 
the basis of an input image as regards images picked under different pose 
5 and illumination conditions every object, even when only a small number of 
reference images are available. 

Another object of the present invention is to provide an image 
matching system, an image matching method, and an image matching 
program that make it possible to conduct matching with a small number of 
10 reference images of three-dimensional object models without conducting 
processing such as converting the input image or the reference image so as 
to make the pose coincide and that makes it possible to conduct matching 
even if a region common to the images is not present. 

Another object of the present invention is to provide an image 
1 5 matching system, an image matching method, and an image matching 
program that make it possible to conduct image matching without always 
generating a certain necessary number of three-dimensional objects with 
respect to all objects. 

Still another object of the present invention is to provide an image 
20 matching system, an image matching method, an image matching program 
that make it possible to conduct retrieval at high speed even when a 
database has reference images concerning a large number of objects 
registered therein. 

The present invention provides an image matching system for 
25 retrieving a reference image similar to an input image, the image matching 
system including means for making a match between the input image and a 
plurality of representative three-dimensional object models, means for 
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making a match between the reference image and the representative three- 
dimensional object models, and means for retrieving the reference image 
similar to the input image by using a result of the match between the input 
image and the representative three-dimensional object models and a result 

5 of the match between the reference image and the representative three- 
dimensional object models. 

The image matching system may further include means for finding a 
reference three-dimensional object model associated with the reference 
image similar to the input image, and means for newly retrieving the 

10 reference image similar to the input image by using the reference three- 
dimensional object model and the input image. 

The image matching system may further include means for finding a 
reference three-dimensional object model associated with the reference 
image similar to the input image, conversion means for squaring an input 

1 5 condition of the input image with that of the reference image by converting 
the input image and/or the reference image on the basis of the reference 
three-dimensional object model, and means for retrieving the reference 
image associated with the input image by making a match between the 
input image and reference image squared in input condition. 

20 In the image matching system, the conversion means may 

previously convert the reference image, and square an input condition of the 
input image with that of the reference image. 

The image matching system may include image input means for 
inputting the input image, a representative three-dimensional object model 

25 storage section for storing a plurality of representative three-dimensional 
object models, image generation means for generating at least one 
comparison image close in input condition to the input image every 
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representative three-dimensional object model on the basis of the 
representative three-dimensional object models stored in the representative 
three-dimensional object model storage section, image matching means for 
calculating a similarity between the input image and each of the comparison 
5 images generated by the image generation means, selecting a maximum 
similarity with respect to comparison images associated with each 
representative three-dimensional object model, and regarding the maximum 
similarity as a similarity between the input image and the representative 
three-dimensional object model, a reference image storage section for 

10 storing the reference images of objects, a reference image matching result 
storage section for storing similarities between the reference images stored 
in the reference image storage section and representative three-dimensional 
object models stored in the representative three-dimensional object model 
storage section, and result matching means for extracting the reference 

1 5 image similar to the input image on the basis of similarities between the 
input image and the representative three-dimensional object models 
calculated by the image matching means and similarities between the 
reference images and the representative three-dimensional object models 
stored in the reference image matching result storage section. 

20 The image matching system may further include three-dimensional 

object model registration means for registering representative three- 
dimensional object models in the representative three-dimensional object 
model storage section, reference image registration means for registering 
reference images in the reference image storage section, and reference 

25 image matching result update means for conducting calculation of the 
similarities using the image matching means, on a combination of a 
reference image and a representative three-dimensional object model newly 
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generated by registration, when a new representative three-dimensional 
object model is registered in the representative three-dimensional object 
model storage section by the three-dimensional object model registration 
means, or when a new reference image is registered in the reference image 
5 storage section by the reference image registration means, and adding a 
result of the calculation to results in the reference image matching result 
storage section. 

In the image matching system, the image matching means may 
calculate a similarity between the input image and a representative three- 

10 dimensional object model every partial region, the reference image matching 
result storage section may store similarities between the reference images 
stored in the reference image storage section and representative three- 
dimensional object models stored in the representative three-dimensional 
object model storage section, every partial region, and the result matching 

1 5 means may extract the reference image similar to the input image on the 
basis of similarities between the input image and the representative three- 
dimensional object models calculated by the image matching means every 
partial region and similarities between the reference images and the 
representative three-dimensional object models, stored in the reference 

20 image matching result storage section every partial region. 

In the image matching system, the result matching means may 
calculate similarities between similarities between the input image and the 
representative three-dimensional object models and similarities between the 
reference images and the representative three-dimensional object models, 

25 and in the calculation provide the resultant similarities with weights on the 
basis of candidate precedence of similarities between the input image and 
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the comparison images and the representative three-dimensional object 
models. 

The image matching system may include image input means for 
inputting the input image, a representative three-dimensional object model 
5 storage section for storing a plurality of representative three-dimensional 
object models, image generation means for generating at least one 
comparison image close in input condition to the input image every 
representative three-dimensional object model on the basis of the 
representative three-dimensional object models stored in the representative 

10 three-dimensional object model storage section, image matching means for 
calculating a similarity between the input image and each of the comparison 
images generated by the image generation means, selecting a maximum 
similarity with respect to comparison images associated with each 
representative three-dimensional object model, and regarding the maximum 

1 5 similarity as a similarity between the input image and the representative 
three-dimensional object model, a reference image storage section for 
storing the reference images of objects, a reference image matching result 
storage section for storing similarities between the reference images stored 
in the reference image storage section and representative three-dimensional 

20 object models stored in the representative three-dimensional object model 
storage section, result matching means for extracting the reference image 
similar to the input image on the basis of similarities between the input 
image and the representative three-dimensional object models calculated by 
the image matching means and similarities between the reference images 

25 and the representative three-dimensional object models stored in the 
reference image matching result storage section, a reference three- 
dimensional object model storage section for storing reference three- 
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dimensional object models associated with the reference images stored in 
the reference image storage section, second image generation means for 
obtaining reference three-dimensional object models associated with 
reference images extracted from the result matching means, from the 
5 reference three-dimensional object model storage section, and generating at 
least one second comparison image close in input condition to the input 
image every reference three-dimensional object model on the basis of the 
obtained reference three-dimensional object models, and second image 
matching means for calculating similarities between the input image and 

10 second comparison images generated by the second image generation means, 
selecting a maximum similarity from among second comparison images 
associated with each of the reference three-dimensional object models, and 
regarding the maximum similarity as a similarity between the input image 
and the reference three-dimensional object model. 

1 5 The image matching may further include three-dimensional object 

model registration means for registering representative three-dimensional 
object models in the representative three-dimensional object model storage 
section, reference image registration means for registering reference images 
in the reference image storage section, and reference image matching result 

20 update means for conducting calculation of the similarities using the image 
matching means, on a combination of a reference image and a 
representative three-dimensional object model newly generated by 
registration, when a new representative three-dimensional object model is 
registered in the representative three-dimensional object model storage 

25 section by the three-dimensional object model registration means, or when a 
new reference image is registered in the reference image storage section by 
the reference image registration means, and adding a result of the 
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calculation to results in the reference image matching result storage section, 
and three-dimensional object model generation means responsive to 
registration of a similarity between the reference image and the 
representative three-dimensional object model in the reference image 
5 matching result storage section conducted by the reference image matching 
result update means, for generating the reference three-dimensional object 
model associated with the reference image by combining the representative 
three-dimensional object models stored in the representative three- 
dimensional object model storage section on the basis of the similarity, and 

10 registering the generated reference three-dimensional object model in the 
reference three-dimensional object model storage section. 

In the image matching system, the three-dimensional object model 
generation means may generate a reference three-dimensional object model 
associated with each reference image by combining representative three- 

15 dimensional object models stored in the representative three-dimensional 
object model storage section every partial region, on the basis of similarities 
obtained every partial region between reference images stored in the 
reference image storage section and representative three-dimensional object 
models stored in the representative three-dimensional object model storage 

20 section, and register the generated reference three-dimensional object model 
in the reference three-dimensional object model storage section. 

In the image matching system, the image matching means may 
calculate a similarity between the input image and a representative three- 
dimensional object model every partial region, the reference image matching 

25 result storage section may store similarities between the reference images 
stored in the reference image storage section and representative three- 
dimensional object models stored in the representative three-dimensional 
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object model storage section, every partial region, and the result matching 
means may extract the reference image similar to the input image on the 
basis of similarities between the input image and the representative three- 
dimensional object models calculated by the image matching means every 
5 partial region and similarities between the reference images and the 
representative three-dimensional object models, stored in the reference 
image matching result storage section every partial region. 

In the image matching system, the result matching means may 
calculate similarities between similarities between the input image and the 

10 representative three-dimensional object models and similarities between the 
reference images and the representative three-dimensional object models, 
and in the calculation, provide the resultant similarities with weights on the 
basis of candidate precedence of similarities between the input image and 
the comparison images and the representative three-dimensional object 

1 5 models. 

The image matching system may include image input means for 
inputting the input image, a representative three-dimensional object model 
storage section for storing a plurality of representative three-dimensional 
object models, image generation means for generating at least one 

20 comparison image close in input condition to the input image every 
representative three-dimensional object model on the basis of the 
representative three-dimensional object models stored in the representative 
three-dimensional object model storage section, image matching means for 
calculating a similarity between the input image and each of the comparison 

25 images generated by the image generation means, selecting a maximum 
similarity with respect to comparison images associated with each 
representative three-dimensional object model, and regarding the maximum 
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similarity as a similarity between the input image and the representative 
three-dimensional object model, a reference image storage section for 
storing the reference images of objects, a reference image matching result 
storage section for storing similarities between the reference images stored 
5 in the reference image storage section and representative three-dimensional 
object models stored in the representative three-dimensional object model 
storage section, result matching means for extracting the reference image 
similar to the input image on the basis of similarities between the input 
image and the representative three-dimensional object models calculated by 

10 the image matching means and similarities between the reference images 
and the representative three-dimensional object models stored in the 
reference image matching result storage section, a reference three- 
dimensional object model storage section for storing reference three- 
dimensional object models associated with the reference images stored in 

1 5 the reference image storage section, image conversion means for obtaining 
reference three-dimensional object models associated with reference images 
extracted from the result matching means, from the reference three- 
dimensional object model storage section, squaring an input condition of the 
input image with that of the reference image extracted by the result 

20 matching means by converting the input image and/or the reference image 
extracted by the result matching means, on the basis of the obtained 
reference three-dimensional object models, and generating partial images 
respectively of the input image and the reference image squared in input 
condition with each other, and partial image matching means for calculating 

25 a similarity between the partial image of the input image and the partial 
image of the reference image generated by the image conversion means. 
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The image matching system may further include three-dimensional 
object model registration means for registering representative three- 
dimensional object models in the representative three-dimensional object 
model storage section, reference image registration means for registering 
5 reference images in the reference image storage section, and reference 
image matching result update means for conducting calculation of the 
similarities using the image matching means, on a combination of a 
reference image and a representative three-dimensional object model newly 
generated by registration, when a new representative three-dimensional 

10 object model is registered in the representative three-dimensional object 
model storage section by the three-dimensional object model registration 
means, or when a reference image is registered in the reference image 
storage section by the reference image registration means, and adding a 
result of the calculation to results in the reference image matching result 

15 storage section, and three-dimensional object model generation means 

responsive to registration of a similarity between the reference image and 
the representative three-dimensional object model in the reference image 
matching result storage section conducted by the reference image matching 
result update means, for generating the reference three-dimensional object 

20 model associated with the reference image by combining the representative 
three-dimensional object models stored in the representative three- 
dimensional object model storage section on the basis of the similarity, and 
registering the generated reference three-dimensional object model in the 
reference three-dimensional object model storage section. 

25 In the image matching system, the three-dimensional object model 

generation means may generate a reference three-dimensional object model 
associated with each reference image by combining representative three - 
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dimensional object models stored in the representative three-dimensional 
object model storage section every partial region, on the basis of similarities 
obtained every partial region between reference images stored in the 
reference image storage section and representative three-dimensional object 
5 models stored in the representative three-dimensional object model storage 
section, and register the generated reference three-dimensional object model 
in the reference three-dimensional object model storage section. 

In the image matching system, the image matching means may 
calculate a similarity between the input image and a representative three- 

10 dimensional object model every partial region, the reference image matching 
result storage section may store similarities between the reference images 
stored in the reference image storage section and representative three- 
dimensional object models stored in the representative three-dimensional 
object model storage section, every partial region, and the result matching 

1 5 means may extract the reference image similar to the input image on the 
basis of similarities between the input image and the representative three- 
dimensional object models calculated by the image matching means every 
partial region and similarities between the reference images and the 
representative three-dimensional object models, stored in the reference 

20 image matching result storage section every partial region. 

In the image matching system, the result matching means may 
calculate similarities between similarities between the input image and the 
representative three-dimensional object models and similarities between the 
reference images and the representative three-dimensional object models, 

25 and in the calculation, provide the resultant similarities with weights on the 
basis of candidate precedence of similarities between the input image and 
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the comparison images and the representative three-dimensional object 
models. 

In the image matching system, the object may be a human face. 
According to the present invention, effects described hereafter are 
5 achieved. 

A first effect will now be described. With respect to an input image 
of an object picked up under a different input condition such as a different 
pose and illumination condition a reference image of the same object can be 
retrieved, even if only one reference image or a small number of reference 

10 images are present. Furthermore, matching can be conducted with a small 
number of reference images of three-dimensional object models, without 
conducting processing such as converting the input image or the reference 
image so as to make their poses coincide with each other. In addition, 
matching can be conducted even if a region common to the images is not 

1 5 present. Furthermore, image matching becomes possible without always 
generating a predetermined number of three-dimensional object models for 
every object. 

The reason is that a reference image is retrieved by comparing a 
result of matching between the input image and representative three- 
20 dimensional object models with a result of matching between reference 
images and representative three-dimensional object models. The reason is 
also that a reference three-dimensional object model is generated by 
combining representative three-dimensional object models and subjected to 
matching. 

25 A second effect is that a reference image can be retrieved at high 

speed with respect to an input image. 
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The reason is that matching of representative three-dimensional 
object models less than the objects is conducted and image matching is 
conducted by using the calculation of similarity of the matching result. Even 
when conducting matching with a reference three-dimensional object model, 
5 reference images having high similarity are extracted by using 

representative three-dimensional object models, and then matching of high 
precedence candidates with reference three-dimensional object models is 
conducted. 

BRIEF DESCRIPTION OF DRAWINGS 
10 FIG. 1 is a block diagram showing a configuration of an image 

matching system according to a first embodiment of the present invention; 

FIG. 2 is a flow char showing operation conducted at the time of 
matching in a first embodiment; 

FIG. 3 is a diagram showing a concrete example of a representative 
1 5 three-dimensional object model in a first embodiment; 

FIG. 4 is a diagram showing concrete examples of a reference image 
in a first embodiment; 

FIG. 5 is a diagram showing a concrete example of a reference image 
matching result in a first embodiment; 
20 FIG. 6 is a diagram showing a concrete example of an input image in 

a first embodiment; 

FIG. 7 is a diagram showing a concrete example of an input image 
matching result in a first embodiment; 

FIG. 8 is a diagram showing a concrete example of result matching 
25 in a first embodiment; 
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FIG. 9 is a block diagram showing a configuration of an image 
matching system according to a second embodiment of the present 
invention; 

FIG. 10 is a flow chart showing operation conducted at the time of 
5 three-dimensional object model registration in a second embodiment; 

FIG. 11 is a flow chart showing operation conducted at the time of 
reference image registration in a second embodiment; 

FIG. 12 is a diagram showing a concrete example of a matching 
result of a three-dimensional object model registered in a second 
10 embodiment; 

FIG. 13 is a diagram showing a concrete example of update of a 
reference image matching result in a second embodiment; 

FIG. 14 is a diagram showing a concrete example of a matching 
result of a registered reference image in a second embodiment; 
1 5 FIG. 15 is a diagram showing a concrete example of update of a 

reference image matching result in a second embodiment; 

FIG. 16 is a block diagram showing a configuration of an image 
matching system according to a third embodiment of the present invention; 
FIG. 17 is a flow chart showing operation conducted at the time of 
20 matching in a third embodiment; 

FIG. 18 is a diagram showing concrete examples of a reference 
three-dimensional object model in a third embodiment; 

FIG. 19 is a diagram showing a concrete example of a reference 
image matching result in a third embodiment; 
25 FIG. 20 is a block diagram showing a configuration of an image 

matching system according to a fourth embodiment of the present invention! 
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FIG. 21 is a flow chart showing operation conducted at the time of 
three-dimensional object model registration in a fourth embodiment; 

FIG. 22 is a flow chart showing operation conducted at the time of 
reference image registration in a fourth embodiment; 
5 FIG. 23 is a block diagram showing a configuration of an image 

matching system according to a fifth embodiment of the present invention; 

FIG. 24 is a flow chart showing operation conducted at the time of 
matching in a fifth embodiment; 

FIG. 25 is a block diagram showing a configuration of an image 
1 0 matching system according to a sixth embodiment of the present invention; 
FIG. 26 is a block diagram showing a configuration of an image 
matching system according to a first conventional technique; 

FIG. 27 is a diagram showing a concrete example of coordinates of a 
three-dimensional object model; 
1 5 FIG. 28 is a block diagram showing a configuration of an image 

matching system according to a second conventional technique; and 

FIG. 29 is a diagram showing a concrete example of a partial region 
of a second conventional technique. 

BEST MODE FOR CARRYING OUT THE INVENTION 
20 Hereafter, embodiments of the present invention will be described in 

detail with reference to the drawings. 

With reference to FIG. 1, an image matching system according to a 
first embodiment of the present invention includes an image input section 
10, an image generation section 30, an image matching section 40, a result 
25 matching section 60, a result display section 80, a reference image storage 
section 70, a representative three-dimensional object model storage section 
20, and a reference image matching result storage section 50. 
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Representative three-dimensional object models (three-dimensional 
shapes of objects and textures on object surfaces) are previously registered 
in the representative three-dimensional object model storage section 20. The 
three-dimensional object models can be generated by using, for example, a 

5 three-dimensional shape measurement apparatus described in Japanese 
Patent Application Laid-Open No. 2001-12925 or an apparatus that 
reconstructs a three-dimensional shape from a plurality of images picked up 
with a large number of cameras and that is described in Japanese Patent 
Application Laid-Open No. 9-91436. 

1 0 As shown in FIG. 27, the three-dimensional object model has a 

shape Pq(x, y, z) and a texture Tq(R, G, B) in a three-dimensional space (x, y, 
z) at an object surface, as information. Q represents an index of a point on 
the object surface, and corresponds to coordinates of a point Q(s,t) obtained 
by projecting a point on the object surface onto a sphere centering around 

1 5 the center of gravity of the object from the center of gravity. For the purpose 
of matching, CG images for learning under various illumination conditions 
are previously generated by using three-dimensional object models and 
computer graphics, and a basic image group is derived by conducting 
principal component analysis on the CG images for learning. 

20 The image generation section 30 generates a plurality of comparison 

images close in illumination condition to an input image obtained from the 
image input section 10, while supposing a pose condition on the basis of a 
representative three-dimensional object model obtained from the 
representative three-dimensional object model storage section 20. Here, the 

25 generation of the comparison image close in illumination condition to the 
input image can be implemented by conducting coordinate conversion on a 
basic image group obtained beforehand on the basis of the supposed pose 
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condition and obtaining coefficients of a linear sum by using the least 
square method so as to make the linear sum of the basic images subjected to 
the coordinate conversion close to the input image. 

A technique for generating a comparison image close to the input 
5 image from the three-dimensional object model is described in, for example, 
"Face matching using automatic correction of both illumination condition 
and pose," Technical Report of the Institute of Electronics, Information and 
Communication Engineers, Vol. 101, No. 524, PRMU200M53-175 (2001), 
pp. 59-64. 

10 The image matching section 40 estimates the pose by comparing the 

input image obtained from the image input section 10 with each of 
comparison images obtained from the image generation section 30, 
calculating a similarity between the input image and each of the comparison 
images, and selecting a comparison image having the greatest similarity for 

1 5 each object. 

The image generation section 30 and the image matching section 40 
handles each of the reference images stored in the reference image storage 
section 70 serving as a database (DB) that stores reference images as the 
input image and matches representative three-dimensional object models 

20 stored in the representative three-dimensional object model storage section 
20 with each of the reference images. Results obtained by the matching are 
previously stored in the reference image matching result storage section 50. 

The result matching section 60 compares a result of matching of the 
input image obtained from the image input section 10 conducted by the 

25 image generation section 30 and the image matching section 40 with the 
matching result of each reference image in the reference image matching 
result storage section 50, and extracts reference images having similar 
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matching results, in the descending order of the similarity. The result 
display section 80 displays an object having the greatest similarity as a 
matching result. 

Reference images which are two-dimensional images of an object to 
5 be retrieved are registered in the reference image storage section 70. As for 
the reference images, there are no restrictions to the input condition 
including the illumination and pose. At least one image is registered every 
object (retrieval subject). 

A plurality of representative three-dimensional object models are 
10 stored in the representative three-dimensional object model storage section 
20. 

General operation of the first embodiment will now be described in 
detail with reference to FIG. 1 and a flow chart shown in FIG. 2. 

At the time of input image matching, an input image is first 

1 5 obtained by the input image section 10 (step 100 in FIG. 2). Subsequently, 
the image generation section 30 generates comparison images that are close 
in the input condition such as the pose and illumination to the input image, 
i.e., comparison images that facilitate comparison, with respect to each of 
the representative three-dimensional object models stored in the 

20 representative three-dimensional object model storage section 20 (step 101). 

The image matching section 40 finds similarity between the input 
image and each of the comparison images (step 102). The result matching 
section 60 calculates a similarity between the matching result and a 
matching result of each of the reference image stored in the reference image 

25 matching result storage section 50, and extracts reference images having 

similar matching results, in the descending order of the similarity (step 103). 
Finally, a reference image having high similarity is displayed (step 104). 
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Effects of the first embodiment which has the configuration and 
which operates as described above will now be described. 

The first embodiment has the configuration in which reference 
images are retrieved by comparing a result of matching between the input 
5 image and the representative three-dimensional object models with a result 
of matching between reference images and the representative three- 
dimensional object models. Even when only one reference image or a small 
number of reference images are present every object, therefore, reference 
images can be retrieved with respect to an input image of an object picked 

10 up under a different condition concerning the pose and illumination. 

The present embodiment has a configuration in which image 
matching is conducted by conducting matching with representative three- 
dimensional object models which are less than objects and conducting 
similarity calculation on the results of the matching. This makes fast 

1 5 retrieval possible. Since the time taken for the similarity calculation of 

matching results is shorter than the time taken for the image matching, the 
retrieval time depends on the number of image matching operations. For 
example, if the number N of the representative three-dimensional object 
models is N = M/lOO where M is the number of objects (the number of 

20 reference images), then the number of required image matching operations 
is L x N = L x M/100 where L is the number of comparison images for each 
representative three-dimensional object model generated in the image 
generation section 30. Thus, retrieval can be conducted with the number of 
times of matching as small as 1/100 of that in the conventional technique. 

25 Operation of the first embodiment will now be described with 

reference to FIGS. 3 to 8 which show concrete embodiments. 
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As shown in FIG. 3, N representative three-dimensional object 
models C, (j = 1, 2, ... , N) are stored in the representative three-dimensional 
object model storage section 20. As shown in FIG. 4, M reference images Ri 
(i = 1, 2, ... , M) respectively of the objects are stored in the reference image 
5 storage section 70 (A plurality of reference images may be present for each 
of the objects. In the ensuing description, however, it is supposed that one 
reference image is stored every object). 

As shown in FIG. 5, a result (similarity Sij) of matching of each of 
the reference images Ri with the representative three-dimensional object 
10 model Cj is stored in the reference image matching result storage section 50 
by processing at the time of reference image registration. (In FIG. 5, the 
matching results are shown in the descending order of the similarity. As a 
matter of fact, however, the matching results may be stored in the order of 
model.) 

1 5 It is supposed that an input image I(u, v) as shown in FIG. 6 is 

obtained by the image input section 10 at the time of matching of the input 
image (step 100 in FIG. 2). Subsequently, the image generation section 30 
generates L comparison images Gjk(u, v) (j = 1, ... , N, k = 1, ... , L) which are 
close in input condition such as the pose and illumination to the input image, 

20 with respect to each of the representative three-dimensional object models 
Cj (j = 1, ... , N) in the representative three-dimensional object model storage 
section 20 (step 101). 

In addition, the image matching section 40 finds a similarity S(I, 
Gjk) between the input image I(u, v) and each of the comparison images 

25 Gjk(u, v), and finds a maximum similarity SojJ= maxk S(I, Gjk) every 

representative three-dimensional object model (step 102). The matching 
results (similarities) Soj become, for example, as shown in FIG. 7. 
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The result matching section 60 calculates a similarity Di = D(Soj, Sij) 
between the matching result Sqj and the matching result Sy of each of the 
reference images in the reference image matching result storage section 50, 
and extracts reference images in the descending order of the similarity Di of 
5 the matching result (step 103). The result of the extraction becomes, for 
example, as shown in FIG. 8. As reference images having a high possibility 
of being an image of the same object as the input image, Ri, Rs and R2 are 
obtained in the cited order. Finally, the reference images having high 
similarities are displayed (step 104). 

1 0 As the calculation method of the similarity Di(Soj, Sij) of the 

matching result, normalized correlation, rank.correlation, or the like can be 
used. The rank correlation is correlation of candidate precedence of the 
matching result. Denoting the candidate precedence of the matching result 
Soj of the input image by Aoj, it follows that Ao,2 = 1, A0.6 = 2 and Ao,3 = 3 in 

1 5 the case of the matching result shown in FIG. 7. Denoting candidate 

precedence of the matching result Sij of each of the reference images by Ay, 
for example, the Spearman's rank correlation can be obtained according to 
the expression 1 - 6E j(A 0j -Ay) 2 /{N(N 2 -l)}. 

In the similarity calculation, the similarities may be calculated after 

20 conducting variable conversion on the variables (such as Soj, Sij and Aoj, Ay). 
The similarities may be calculated by weighting variables with weights g(Aoj, 
Ay) based on the candidate precedence Aoj and/or Ay. For example, specific 
gravities of high precedence candidates become great by setting g(Aoj, Ay) = 
l/(Aoj+Aij) and replacing the similarities Soj and Sy respectively by 

25 Soj/(Aoj+Ay) and Sy /(Aoj+Ay). The similarities may be calculated with low 
precedence candidates excluded. 



25 



An image matching system according to a second embodiment of the 
present invention will now be described with reference to FIGS. 9 to 15. An 
image matching system according to the second embodiment of the present 
invention includes an image input section 10, an image generation section 
5 30, an image matching section 40, a result matching section 60, a result 
display section 80, a reference image storage section 70, a reference image 
registration section 75, a representative three-dimensional object model 
storage section 20, a three-dimensional object model registration section 25, 
a reference image matching result storage section 50, and a reference image 

10 matching result update section 55. This configuration is obtained by adding 
the reference image registration section 75, the three-dimensional object 
model registration section 25, and the reference image matching result 
update section 55 to the configuration of the first embodiment. 

As to components in the image matching system according to the 

1 5 second embodiment that are the same as those in the first embodiment, 
description will be omitted. The components added in the present 
embodiment will be described. 

The three-dimensional object model registration section 25 registers 
a new representative three-dimensional object model (a three-dimensional 

20 shape of an object and a texture on the object surface) in the representative 
three-dimensional object model storage section 20. 

When at the time of registration a representative three-dimensional 
object model is registered in the representative three-dimensional object 
model storage section 20 by the three-dimensional object model 25, and 

25 when a reference image is registered in the reference image storage section 
70 by the reference image registration section 75, the reference image 
matching result update section 55 conducts matching operation on a 



n II 

combination of a new reference image and a representative three- 
dimensional object model by using the image generation section 30 and the 
image matching section 40, and adds a result of the matching to data in the 
reference image matching result storage section 50. 
5 The reference image registration section 75 registers reference 

images which are two-dimensional images of an object to be retrieved, in the 
reference image storage section 70. As for the reference images, there are no 
restrictions to the input condition including the illumination and pose. At 
least one image is registered every object (retrieval subject). 

10 By the way, the three-dimensional object model registration section 

25 is the same as the three-dimensional object model registration section 25 
in the second conventional technique shown in FIG. 28. Representative 
three-dimensional object models obtained from the three-dimensional object 
model registration section 25 are previously stored in the representative 

1 5 three-dimensional object model storage section 20. 

General operation of the second embodiment will now be described 
in detail with reference to FIG. 9 and flow charts shown in FIGS. 2, 10 and 
11. 

Operation conducted at the time of matching of the input image is 
20 completely the same as the operation of the first embodiment shown in FIG. 
2. 

At the time of input image matching, an input image is first 
obtained by the input image section 10 (step 100 in FIG. 2). Subsequently, 
the image generation section 30 generates comparison images that are close 
25 in the input condition such as the pose and illumination to the input image, 
i.e., comparison images that facilitate comparison, with respect to each of 
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the representative three-dimensional object models stored in the 
representative three-dimensional object model storage section 20 (step 101). 

The image matching section 40 finds a similarity between the input 
image and each of the comparison images (step 102). The result matching 
5 section 60 calculates a similarity between the matching result and the 
matching result of each of the reference images stored in the reference 
image matching result storage section 50, and extracts reference images 
having similar matching results, in the descending order of the similarity 
(step 103). Finally, a reference image having high similarity is displayed 
10 (step 104). 

Operation conducted at the time of representative three-dimensional 
object model registration and operation conducted at the time of reference 
image registration will now be described. 

At the time of representative three-dimensional object model (a 
1 5 three-dimensional shape of an object and a texture on the object surface) 

registration, the three-dimensional object model registration section 25 first 
registers a new representative three-dimensional object model in the 
representative three-dimensional object model storage section 20 (step 200 
in FIG. 10). 

20 Subsequently, the reference image matching result update section 

55 sends each of the reference images stored in the reference image storage 
section 70 to the image input section 10 as the input image. The reference 
image matching result update section 55 conducts matching, in the image 
matching section 40, of each of the reference images with comparison 

25 images generated by the image generation section 30 on the basis of the 
registered representative three-dimensional object model, and finds a 
similarity (step 201). Finally, the reference image matching result update 
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section 55 adds a result of the matching to each of the matching results of 
the reference images stored in the reference image matching result storage 
section 50 (step 202). 

At the time of reference image registration, the reference image 

5 registration section 75 first registers a new reference image in the reference 
image storage section 70 (step 210 in FIG. 11). 

Subsequently, the reference image matching result update section 
55 sends the reference image registered in the reference image storage 
section 70 to the image input section 10 as the input image. The reference 

10 image matching result update section 55 conducts matching, in the image 
matching section 40, of the reference image with comparison images 
generated by the image generation section 30 on the basis of the 
representative three-dimensional object model stored in the representative 
three-dimensional object model storage section 20, and finds similarities 

15 (step 211). Finally, the reference image matching result update section 55 
adds a result of the matching to the reference image matching result storage 
section 50 (step 212). 

Effects of the second embodiment which has the configuration and 
which operates as described above will now be described. 

20 The second embodiment has the configuration in which reference 

images are retrieved by comparing a result of matching between the input 
image and the representative three-dimensional object models with a result 
of matching between reference images and the representative three- 
dimensional object models. Even when only one reference image or a small 

25 number of reference images are present every object, therefore, reference 
images can be retrieved with respect to an input image of an object picked 
up under a different condition concerning the pose and illumination. 
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The present embodiment has a configuration in which image 
matching is conducted by conducting matching with representative three- 
dimensional object models which are less than objects and conducting 
similarity calculation on the results of the matching. This makes fast 
5 retrieval possible. Since the time taken for the similarity calculation of 

matching results is shorter than the time taken for the image matching, the 
retrieval time depends on the number of image matching operations. For 
example, if the number N of the representative three-dimensional object 
models is N = M/100 where M is the number of objects (the number of 

10 reference images), then the number of required image matching operations 
is L x N = L x M/100 where L is the number of comparison images for each 
representative three-dimensional object model generated in the image 
generation section 30. Thus, retrieval can be conducted with the number of 
times of matching as small as 1/100 of that in the conventional technique. 

1 5 Operation of the second embodiment will now be described with 

reference to concrete embodiments. 

As shown in FIG. 3, N representative three -dimensional object 
models Cj (j = 1, 2, ... , N) are stored in the representative three-dimensional 
object model storage section 20 in the same way as the first embodiment. As 

20 shown in FIG. 4, M reference images Ri (i = 1, 2, ... , M) of the objects are 
stored in the reference image storage section 70. As shown in FIG. 5, a 
result (similarity Sij) of matching of each reference image Ri with the 
representative three-dimensional object model Cj is stored in the reference 
image matching result storage section 50 by processing at the time of 

25 reference image registration. 

It is supposed that an input image I(u, v) as shown in FIG. 6 is 
obtained by the image input section 10 at the time of matching of the input 
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image (step 100 in FIG. 2). Subsequently, the image generation section 30 
generates L comparison images Gjk(u, v) (j = 1, ... , N, k = 1, ... , L) which are 
close in input condition such as the pose and illumination to the input image, 
with respect to each of the representative three-dimensional object models 
5 Q (j = 1, ... , N) in the representative three-dimensional object model storage 
section 20 (step 101). 

In addition, the image matching section 40 finds a similarity S(I, 
Gjk) between the input image I(u, v) and each of the comparison images 
Gjk(u, v), and finds a maximum similarity S 0 j = maxk S(I, Gjk) every 

10 representative three-dimensional object model (step 102). The matching 
results (similarities) S 0 j become, for example, as shown in FIG. 7. 

The result matching section 60 calculates similarities Di = D(Soj, Sij) 
between the matching result Soj and the matching results Sij of the reference 
images in the reference image matching result storage section 50, and 

15 extracts reference images in the descending order of the similarity Di of the 
matching result (step 103). The result of the extraction becomes, for 
example, as shown in FIG. 8. As reference images having a high possibility 
of being an image of the same object as the input image, Ri, Rs and R2 are 
obtained in the cited order. Finally, the reference images having high 

20 similarities are displayed (step 104). 

At the time of representative three-dimensional object model 
registration, the three-dimensional object model registration section 25 first 
registers a new representative three-dimensional object model. If N=50 
representative three-dimensional object models are already registered in the 

25 representative three-dimensional object model storage section 20, the three- 
dimensional object model registration section 25 registers a new fifty-first 
representative three-dimensional object model C51 (step 200 in FIG. 10). 
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Subsequently, the reference image matching result update section 
55 sends each reference image Ri stored in the reference image storage 
section 70 to the image input section 10 as the input image. The reference 
image matching result update section 55 conducts matching of the each 
5 reference image Ri with the registered representative three-dimensional 
object model C51 by using the image generation section 30 and the image 
matching section 40, and finds each similarity Si,5i = maxkS(Ri, G5i,k) (step 
201). 

The matching result (similarity) Si,si becomes, for example, as shown 

10 in FIG. 12. Finally, as shown in FIG. 13, the reference image matching 

result update section 55 adds the matching result to the matching results of 
each of the reference images stored in the reference image matching result 
storage section 50 (step 202). 

At the time of reference image registration, the reference image 

1 5 registration section 75 first registers a new reference image in the reference 
image storage section 70. If M=100 reference images are already registered 
in the reference image storage section 70, the reference image registration 
section 75 registers a new hundred first reference image R101 in the 
reference image storage section 70 (step 210 in FIG. 11). 

20 Subsequently, the reference image matching result update section 

55 sends the reference image R101 registered in the reference image storage 
section 70 to the image input section 10 as the input image. The reference 
image matching result update section 55 conducts matching of the reference 
image R101 with each three-dimensional object model Q stored in the 

25 representative three-dimensional object model storage section 20 by using 
the image generation section 30 and the image matching section 40, and 
finds each similarity Sioij = maxk S(Rioi, Gjk) (step 211). 



The matching result (similarity) becomes, for example, as shown in 
FIG. 14. Finally, the reference image matching result update section 55 
adds the matching result to the reference image matching result storage 
section 50 (step 212). 
5 An image matching system according to a third embodiment of the 

present invention will now be described in detail with reference to FIGS. 16 
to 19. 

With reference to FIG. 16, an image matching system according to 
the third embodiment of the present invention includes an image input 

10 section 10, an image generation section 30, an image matching section 40, a 
result matching section 60, a second image generation section 31, a second 
image matching section 41, a result display section 80, a reference image 
storage section 70, a representative three-dimensional object model storage 
section 20, a reference image matching result storage section 50, and a 

15 reference three-dimensional object model storage section 21. 

These components nearly operate as described below. The image 
input section 10, the image generation section 30, the image matching 
section 40, the result matching section 60, the result display section 80, the 
reference image storage section 70, and the representative three- 

20 dimensional object model storage section 20 conduct the same processing as 
the processing conducted in the first embodiment of the present invention 
shown in FIG. 1. 

Reference three-dimensional object models associated with the 
reference image are previously stored in the reference three-dimensional 

25 object model storage section 21. The reference three-dimensional object 
models can be generated by combining representative three-dimensional 
object models stored in the representative three-dimensional object model 
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storage section 20 on the basis of information concerning the reference 
image matching result registered in the reference image matching result 
storage section 50. Or if three-dimensional object models of the same object 
as the reference image are previously generated by a three-dimensional 
5 shape measurement apparatus in the same way as the above-described 
representative three-dimensional object model registration, the three- 
dimensional object models may be used. 

The second image generation section 31 generates comparison 
images that are close in input condition such as the pose and illumination 

10 condition to the input image obtained from the image input section 10, for a 
reference image that is a high precedence candidate in the matching results 
obtained from the result matching section 60, on the basis of each of the 
reference three-dimensional object model associated with the reference 
image obtained from the reference three-dimensional object model storage 

15 section 21. 

The second image matching section 41 compares the input image 
obtained from the image input section 10 with each of the comparison 
images obtained from the second image generation section 31, and 
calculates each similarity. 

20 General operation of the third embodiment will now be described in 

detail with reference to FIG. 16 and a flow chart shown in FIG. 17. 

At the time of input image matching, steps 100, 101, 102 and 103 
shown in FIG. 17 are the same as the operation conducted in the first 
embodiment shown in FIG. 2. 

25 The second image generation section 31 generates comparison 

images that are close in input condition such as the pose and illumination 
condition to the input image obtained from the image input section, for a 
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reference image that is a high precedence candidate in the matching results 
obtained from the result matching section 60, on the basis of each of the 
reference three-dimensional object model associated with the reference 
image obtained from the reference three-dimensional object model storage 
5 section 21 (step 111). 

The second image matching section 41 compares the input image 
obtained from the image input section with each of the comparison images 
obtained from the second image generation section 31, and calculates each 
similarity (step 112). Finally, the reference image having high similarity is 

10 displayed (step 104). 

Effects of the third embodiment which has the configuration and 
which operates as described above will now be described. 

The third embodiment has the configuration in which the reference 
three-dimensional object model generated by combining representative 

1 5 three-dimensional object models is matched. Even when only one reference 
image is present every object, therefore, reference images can be retrieved 
by using the reference three-dimensional object models, with respect to an 
input image of an object picked up under a different condition concerning 
the pose and illumination. 

20 The present embodiment has a configuration in which reference 

images having high similarity are extracted by using a representative three- 
dimensional object model and then matching of the reference three- 
dimensional object model with high precedence candidates is conducted. As 
a result, reference images can be retrieved at high speed. 

25 Operation of the third embodiment will now be described with 

reference to concrete embodiments. 
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In the same way as the operation of the first embodiment, 
representative three-dimensional object models Cj (j = 1, 2, ... , N) as shown 
in FIG. 3 are stored in the representative three-dimensional object model 
storage section 20. Reference images Ri (i = 1, 2, ... , M) respectively of the 
5 objects as shown in FIG. 4 are stored in the reference image storage section 
70. A result (similarity) Sij of matching of each of the reference images Ri 
with the representative three-dimensional object model Cj as shown in FIG. 
5 is stored in the reference image matching result storage section 50. 

In addition, M reference three-dimensional object models Bi (i = 1, 
10 2, ... , M) associated with the reference image Ri are previously stored in the 
reference three-dimensional object model storage section 21 as shown in FIG. 
18. 

It is supposed that an input image I(u, v) as shown in FIG. 6 is 
obtained by the image input section 10 at the time of matching of the input 

1 5 image (step 100 in FIG. 16). According to the same processing as the 

operation in the first embodiment, Ri, Rs and R2 are obtained in the cited 
order as reference images having a high possibility of being an image of the 
same object as the input image as shown in FIG. 8 by the image generation 
section 30, the image matching section 40, the result matching section 60, 

20 and the result matching section 60 (steps 101, 102 and 103). 

With respect to, for example, the reference images Ri, Rs and R2 
which are three high precedence candidates in the matching result obtained 
from the result matching section 60, the second image generation section 31 
acquires associated reference three-dimensional object models Bi, B5 and B2 

25 from the reference three-dimensional object model storage section 21, and 
generates comparison images Hjk(u, v) (j = 1, 5, 2, k = 1, ... , L) which are 
close in input condition such as the pose and illumination to the input image 
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obtained from the image input section 10 (step 111). The generation of the 
comparison images Hjk(u, v) is conducted by using a method similar to the 
step S 101. In other words, the second image generation section 31 generates 
L comparison images Hjk(u, v) (j = 1, 5, 2, k = 1, ... , L) which are close in 
5 input condition such as the pose and illumination to the input image, with 
respect to the reference three-dimensional object models Bj (j = 1, 5, 2) in the 
reference three-dimensional object model storage section 21. The second 
image matching section 41 finds a similarity S(I, Hjk) between the input 
image I(u, v) and each comparison image Hjk(u, v), and finds a maximum 

10 similarity S 0 j = maxk S(I, Hjk) every model (step 112). 

The matching results become, for example, as shown in FIG. 19. If 
So5>Soi>So2, then R5, Ri and R2 are obtained in the cited order as reference 
images having a high possibility of being an image of the same object as the 
input image. Finally, the reference images having high similarities are 

15 displayed (step 104). 

An image matching system according to a fourth embodiment of the 
present invention will now be described in detail with reference to FIGS. 20 
to 22. 

With reference to FIG. 20, an image matching system according to 
20 the fourth embodiment of the present invention includes an image input 
section 10, an image generation section 30, an image matching section 40, a 
result matching section 60, a second image generation section 31, a second 
image matching section 41, a result display section 80, a reference image 
storage section 70, a reference image registration section 75, a 
25 representative three-dimensional object model storage section 20, a three- 
dimensional object model registration section 25, a reference image 
matching result storage section 50, a reference image matching result 
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II 

update section 55, a reference three-dimensional object model storage 
section 21, and a three-dimensional object model generation section 27. 

The fourth embodiment has a configuration obtained by adding the 
reference image registration section 75, the three-dimensional object model 
5 registration section 25, the reference image matching result update section 
55 and the three-dimensional object model generation section to the 
configuration of the third embodiment. 

As to components in the image matching system according to the 
fourth embodiment that are the same as those in the third embodiment, 

10 description will be omitted. The components added in the present 
embodiment will be described. 

These components nearly operate as described below. The image 
input section 10, the image generation section 30, the image matching 
section 40, the result matching section 60, the result display section 80, the 

1 5 reference image storage section 70, the reference image registration section 
75, the representative three-dimensional object model storage section 20, 
the three-dimensional object model registration section 25, and the 
reference image matching result update section 55 conduct the same 
processing as the processing conducted in the first embodiment of the 

20 present invention shown in FIG. 1 and the second embodiment shown in 
FIG. 9. 

The reference three-dimensional object model storage section 21, the 
second image generation section 31, and the second image matching section 
41 conduct the same processing as the processing conducted in the third 
25 embodiment shown in FIG. 16. 

When at the time of registration a reference image matching result 
is registered in the reference image matching result storage section 50 by 
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the reference image matching result update section 55, the three- 
dimensional object model generation section 27 generates a reference three- 
dimensional object model associated with the reference image by combining 
representative three-dimensional object models in the representative three- 
5 dimensional object model storage section 20 on the basis of information of 
the reference image matching result, and registers the reference three- 
dimensional object model in the reference three-dimensional object model 
storage section 21, or updates a reference three-dimensional object model in 
the reference three-dimensional object model storage section 21. 

1 0 With respect to the reference images of high precedence candidates 

in the matching result obtained from the result matching section 60, the 
second image generation section 31 generates comparison images close in 
the input condition such as the pose and illumination condition to the input 
image obtained from the image input section 10, on the basis of each of the 

1 5 reference three-dimensional object models associated with the reference 
image obtained from the reference three-dimensional object model storage 
section 21. 

The second image matching section 41 compares the input image 
obtained from the image input section 10 with each of the comparison 
20 images obtained from the second image generation section 31, and 
calculates each similarity. 

General operation of the fourth embodiment will now be described in 
detail with reference to FIG. 20 and flow charts shown in FIGS. 17, 21 and 
22. 

25 At the time of matching of the input image, operation conducted at 

steps 100, 101, 102 and 103 in FIG. 17 is the same as that conducted in the 
first embodiment shown in FIG. 2. 
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The second image generation section 31 generates comparison 
images that are close in input condition such as the pose and illumination 
condition to the input image obtained from the image input section, for a 
reference image that is a high precedence candidate in the matching results 
5 obtained from the result matching section 60, on the basis of each of the 
reference three-dimensional object models associated with the reference 
image obtained from the reference three-dimensional object model storage 
section 21 (step 111). 

The second image matching section 41 compares the input image 

10 obtained from the image input section with each of the comparison images 
obtained from the second image generation section 31, and calculates each 
similarity (step 112). Finally, the reference image having high similarity is 
displayed (step 104). 

At the time of three-dimensional object model registration, operation 

1 5 conducted at steps 200, 201 and 202 in FIG. 21 is the same as that 

conducted in the second embodiment shown in FIG. 10. Finally, the three- 
dimensional object model generation section 27 regenerates a reference 
three-dimensional object model associated with each reference image by 
combining representative three-dimensional object models in the 

20 representative three-dimensional object model storage section 20 on the 
basis of information of each reference image matching result in the 
reference image matching result storage section 50, and registers the 
reference three-dimensional object model in the reference three-dimensional 
object model storage section 21, or replaces a stored reference three - 

25 dimensional object model by it (step 220). 

At the time of reference image registration, operation conducted at 
steps 210, 211 and 212 in FIG. 22 is the same as that conducted in the 
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second embodiment shown in FIG. 11. Finally, the three-dimensional object 
model generation section 27 generates a reference three-dimensional object 
model associated with the reference image by combining representative 
three-dimensional object models in the representative three-dimensional 
5 object model storage section 20 on the basis of information of the reference 
image matching result newly registered in the reference image matching 
result storage section 50, and additionally registers the reference three- 
dimensional object model in the reference three-dimensional object model 
storage section 21 (step 221). 

10 Effects of the fourth embodiment which has the configuration and 

which operates as described above will now be described. 

The fourth embodiment has the configuration in which the reference 
three-dimensional object model generated by combining representative 
three-dimensional object models is matched. Even when only one reference 

1 5 image is present every object, therefore, reference images can be retrieved 
by using the reference three-dimensional object models, with respect to an 
input image of an object picked up under a different condition concerning 
the pose and illumination. 

The present embodiment has a configuration in which reference 

20 images having high similarity are extracted by using a representative three- 
dimensional object model and then matching of the reference three- 
dimensional object model with high precedence candidates is conducted. As 
a result, reference images can be retrieved at high speed. 

Operation of the fourth embodiment will now be described with 

25 reference to concrete embodiments. 

In the same way as the operation of the first embodiment, 
representative three-dimensional object models Gj (j = 1, 2, ... , N) as shown 
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in FIG. 3 are stored in the representative three-dimensional object model 
storage section 20. Reference images Ri (i = 1, 2, ... , M) respectively of the 
objects as shown in FIG. 4 are stored in the reference image storage section 
70. A result (similarity) Sy of matching of each reference image Ri with the 
5 representative three-dimensional object model Cj as shown in FIG. 5 is 
stored in the reference image matching result storage section 50. 

In addition, M reference three-dimensional object models Bi (i = 1, 
2, ... , M) associated with the reference image Ri are previously stored in the 
reference three-dimensional object model storage section 21 by the 
10 processing conducted at the time of reference image registration, as shown 
in FIG. 18. 

It is supposed that an input image I(u, v) as shown in FIG. 6 is 
obtained by the image input section 10 at the time of matching of the input 
image (step 100 in FIG. 16). According to the same processing as the 

1 5 operation in the first embodiment, Ri, Rs and R2 are obtained in the cited 
order as reference images having a high possibility of being an image of the 
same object as the input image as shown in FIG. 8 by the image generation 
section 30, the image matching section 40, and the result matching section 
60 (steps 101, 102 and 103). 

20 With respect to, for example, the reference images Ri, R5 and R2 

which are three high precedence candidates in the matching result obtained 
from the result matching section 60, the second image generation section 31 
acquires associated reference three-dimensional object models Bi, B5 and B2 
from the reference three-dimensional object model storage section 21, and 

25 generates comparison images Hjk(u, v) (j = 1, 5, 2, k = 1, ... , L) which are 

close in input condition such as the pose and illumination to the input image 
obtained from the image input section 10 (step 111). The second image 
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matching section 41 finds a similarity S(I, Hjk) between the input image I(u, 
v) and each comparison image Hjk(u, v), and finds a maximum similarity S 0 j 
= maxk S(I, Hjk) every model (step 112). 

The matching results become, for example, as shown in FIG. 19, and 

5 R5, Ri and R2 are obtained in the cited order as reference images having a 
high possibility of being an image of the same object as the input image. 
Finally, the reference images having high similarity are displayed (step 104). 

At the time of three-dimensional object model registration, the 
three-dimensional object model registration section 25 first registers a new 

10 representative three-dimensional object model in the representative three- 
dimensional object model storage section 20. If N=50 three-dimensional 
object models are already registered in the representative three-dimensional 
object model storage section 20, the three-dimensional object model 
registration section 25 registers a new fifty-first representative three* 

1 5 dimensional object model C51 (step 200 in FIG. 21). 

Subsequently, by the same processing as the operation in the second 
embodiment, the reference image matching result update section 55 updates 
the matching result Sij of each reference image in the reference image 
matching result storage section 50 (steps 201 and 202). 

20 Finally, the three-dimensional object model generation section 27 

regenerates a reference three-dimensional object model Bi associated with 
each reference image Ri (i = 1, 2, ... , M) by combining representative three- 
dimensional object models Cj in the representative three-dimensional object 
model storage section 20 on the basis of information of each reference image 

25 matching result Sij in the reference image matching result storage section 
50, and replaces the reference three-dimensional object model already 
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stored in the reference three-dimensional object model storage section 21 by 
the reference three-dimensional object model Bi (step 220). 

Denoting the shape and texture of the representative three- 
dimensional object model Cj (j = 1, 2, ... , N) respectively by Pqj(x, y, z) and 
5 Tqj(R, G, B) and denoting the shape and texture of the reference three- 
dimensional object model Bi (i = 1, 2, ... , M) respectively by Pqi(x, y, z) and 
Tqi(R, G, B), the reference three-dimensional object model is calculated 
according to, for example, the following expressions. 

P Qi (x, y, z) = S j fCSij) Pqj(x, y, z) 
1 0 T Qi (R, G, B) = I i fCSy) TqjCR, G, B) 

Here, f is a function that monotonously increases as Sy increases, 
and that satisfies the relation 2 j tfCSy) = 1. As the simplest example, f can be 
implemented by f(Sij) = Sij/2 j Sij. 

At the time of reference image registration, the reference image 
1 5 registration section 75 first registers a new reference image in the reference 
image storage section 70. If M=100 reference images are already registered 
in the reference image storage section 70, the reference image registration 
section 75 registers a new hundred first reference image Rioi in the 
reference image storage section 70 (step 210 in FIG. 22). 
20 Subsequently, the reference image matching result update section 

55 adds a matching result Sioi j associated with the reference image Rioi to 
the reference image matching result storage section 50 by conducting the 
same processing as the operation in the second embodiment (steps 211 and 
212). 

25 Finally, the three-dimensional object model generation section 27 

generates a reference three-dimensional object model Bioi associated with 
the reference image Rioi by combining representative three-dimensional 



44 



object models Cj in the representative three-dimensional object model 
storage section 20 on the basis of information of the reference image 
matching result Sioij in the reference image matching result storage section 
50, and adds the reference three-dimensional object model Bioi to the 
5 reference three-dimensional object model storage section 21 (step 221). 

An image matching system according to a fifth embodiment of the 
present invention will now be described in detail with reference to FIGS. 23 
and 24. 

With reference to FIG. 23, an image matching system according to 
10 the fifth embodiment of the present invention includes an image input 

section 10, an image generation section 30, an image matching section 40, a 
result matching section 60, an image conversion section 36, a partial image 
matching section 45, a result display section 80, a reference image storage 
section 70, a representative three-dimensional object model storage section 
15 20, a reference image matching result storage section 50, and a reference 
three-dimensional object model storage section 21. 

These components nearly operate as described below. The image 
input section 10, the image generation section 30, the image matching 
section 40, the result matching section 60, the result display section 80, the 
20 reference image storage section 70, and the representative three- 
dimensional object model storage section 20 conduct the same processing as 
the processing conducted in the first embodiment of the present invention 
shown in FIG. 1. 

With respect to a reference image of a high precedence candidate in 
25 the matching result obtained from the result obtained from the result 
matching section 60, the image conversion section 36 converts the input 
image and/or the reference image so as to make the input condition (such as 
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the pose condition) the same on the basis of each reference three- 
dimensional object model associated with the reference image obtained from 
the reference three-dimensional object model storage section 21. Thus, the 
image conversion section 36 generates partial images. The image conversion 

5 section 36 is similar to the image conversion section 35 in the second 
conventional technique shown in FIG. 28. 

The partial image matching section 45 conducts comparison on the 
partial images of the converted input image and reference image obtained 
from the image conversion section 36, and calculates the similarity. The 

10 similarity calculation is conducted in the same way as the step 102. 

General operation of the fifth embodiment will now be described in 
detail with reference to FIG. 23 and a flow chart shown in FIG. 24. 

At the time of input image matching, steps 100, 101, 102 and 103 
shown in FIG. 24 are the same as the operation conducted in the first 

1 5 embodiment shown in FIG. 2. With respect to a reference image of a high 
precedence candidate in the matching result obtained from the result 
obtained from the result matching section 60, the image conversion section 
36 converts the input image and/or the reference image so as to make the 
input condition (such as the pose condition) the same on the basis of each 

20 reference three-dimensional object model associated with the reference 
image obtained from the reference three-dimensional object model storage 
section 21. Thus, the image conversion section 36 generates partial images 
(step 121). 

The partial image matching section 45 conducts comparison on the 
25 partial images of the converted input image and reference image obtained 
from the image conversion section 36, and calculates the similarity (step 
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122). Finally, the reference image having high similarity is displayed (step 
104). 

In the fifth embodiment of the present invention, the image 
conversion section 36 converts the input image and/or the reference image. 

5 Alternatively, the reference image may be previously converted to that 
under a standard input condition (for example, a standard pose condition) 
and stored, and the image conversion section 36 may convert the input 
image to that under the standard input condition (for example, the standard 
pose condition). By doing so, it becomes unnecessary to convert the reference 

10 image each time the matching is conducted, and the matching time can be 
shortened. 

An image matching system according to a sixth embodiment of the 
present invention will now be described in detail with reference to FIG. 25. 
With reference to FIG. 25, an image matching system according to 

1 5 the sixth embodiment of the present invention includes an image input 

section 10, an image generation section 30, an image matching section 40, a 
result matching section 60, an image conversion section 36, a partial image 
matching section 45, a result display section 80, a reference image storage 
section 70, a reference image registration section 75, a representative three- 

20 dimensional object model storage section 20, a three-dimensional object 
model registration section 25, a reference image matching result storage 
section 50, a reference image matching result update section 55, a reference 
three-dimensional object model storage section 21, and a three-dimensional 
object model generation section 27. 

25 These components nearly operate as described below. The image 

input section 10, the image generation section 30, the image matching 
section 40, the result matching section 60, the result display section 80, the 
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reference image storage section 70, the reference image registration section 
75, the representative three-dimensional object model storage section 20, 
the three-dimensional object model registration section 25, and the 
reference image matching result update section 55 conduct the same 
5 processing as the processing conducted in the first embodiment of the 
present invention shown in FIG. 1 and the second embodiment shown in 
FIG. 9. 

The reference three-dimensional object model storage section 21 and 
the three-dimensional object model generation section 27 conduct the same 
10 processing as the processing conducted in the third embodiment shown in 
FIG. 16 and the fourth embodiment of the present invention shown in FIG. 
20. 

With respect to a reference image of a high precedence candidate in 
the matching result obtained from the result obtained from the result 

1 5 matching section 60, the image conversion section 36 converts the input 

image and/or the reference image so as to make the input condition (such as 
the pose condition) the same on the basis of each reference three- 
dimensional object model associated with the reference image obtained from 
the reference three-dimensional object model storage section 21. Thus, the 

20 image conversion section 36 generates partial images. The partial image 
matching section 45 conducts comparison on the partial images of the 
converted input image and reference image obtained from the image 
conversion section 36, and calculates the similarity. 

General operation of the sixth embodiment will now be described in 

25 detail with reference to FIG. 25 and the flow chart shown in FIG. 24. 

At the time of input image matching, steps 100, 101, 102 and 103 
shown in FIG. 24 are the same as the operation conducted in the first 
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embodiment shown in FIG. 2. With respect to a reference image of a high 
precedence candidate in the matching result obtained from the result 
obtained from the result matching section 60, the image conversion section 
36 converts the input image and/or the reference image so as to make the 
5 input condition (such as the pose condition) the same on the basis of each 
reference three-dimensional object model associated with the reference 
image obtained from the reference three-dimensional object model storage 
section 21. Thus, the image conversion section 36 generates partial images 
(step 121). 

10 The partial image matching section 45 conducts comparison on the 

partial images of the converted input image and reference image obtained 
from the image conversion section 36, and calculates the similarity (step 
122). Finally, the reference image having high similarity is displayed (step 
104). 

1 5 In the sixth embodiment of the present invention, the image 

conversion section 36 converts the input image and/or the reference image. 
Alternatively, the reference image may be previously converted to that 
under a standard input condition (for example, a standard pose condition) 
and stored, and the image conversion section 36 may convert the input 

20 image to that under the standard input condition (for example, the standard 
pose condition). 

When finding the similarity S(I, Gjk) between the input image I(u, v) 
and each of the comparison images Gjk(u, v) in the first to sixth 
embodiments of the present invention, the image matching section 40 finds 
25 one similarity S(I, Gjk) on the whole. Alternatively, the image matching 
section 40 may find similarity S(I, G'jkm) every partial region m, and find a 
maximum similarity S'ojm = maxk S(I, G'jkm) every model. 



49 



The partial regions are, for example, regions as shown in FIG. 29. In 
this case, similarity S'ijm = maxkS(Ri, Gjkm) of each partial region m is stored 
in the reference image matching result storage section 50 as well. The result 
matching section 60 calculates a similarity Di = S m D(S'ojm, S'ijm) between the 
5 matching result S'ojm and the matching result S'ijm of each of the reference 
images in the reference image matching result storage section 50, and 
extracts reference images in the descending order of the similarity Di of the 
matching result. Furthermore, in the three-dimensional object model 
generation section 27 in the fourth and sixth embodiments as well, 
10 representative three-dimensional object models may be combined every 
partial region. 

In the first to sixth embodiments of the present invention, operation 
of retrieving an image of the same object as the input image from among a 
large number of reference images has been described. However, it is also 

1 5 possible to apply the present invention to one-to-one matching for 

determining whether a specific reference image is an image of the same 
object as the input image. 

It is supposed that a specific reference image is Ri. In the first and 
second embodiments, the result matching section 60 calculates a similarity 

20 Di = D(Soj, Sij) between the matching result Soj of the input image and the 
matching result Sij of the reference image Ri. If the similarity Di is greater 
than a threshold, Ri can be judged to be the same object as the input image. 
In the third, fourth, fifth and sixth embodiments, the judgment can be made 
by determining whether the similarity between the input image and a 

25 specific reference image in the second image matching section 41 and the 
partial image matching section 45 is greater than a threshold. 
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In the image matching system according to the present invention, it 
is a matter of course that functions of components can be implemented by 
using hardware. The functions can also be implemented by reading an 
image matching program 500 into a computer to make the computer execute 
5 the functions of components. The image matching program 500 is stored in a 
recording medium such as a magnetic disk or a semiconductor memory. The 
computer reads the image matching program 500 from the recording 
medium. 

INDUSTRIAL APPLICABILITY 
1 0 The present invention can be used for person identification, 

individual authentication, or the like using an image of a face or the like. 
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